An image of a graphical user interface (GUI) may include several GUI elements. Automated testing tools may be interested in understanding interactions between users and GUI elements. Automated testing is one tool that some companies use to ensure that new and/or updated applications function correctly. Automated testing can help designers evaluate an application's responses to various sequences of inputs without requiring a designer to step through sequences manually when the application is modified. Some conventional automated testing tools attempt to model object hierarchies of programming languages and/or communication protocols of applications to determine where and/or how to provide a pre-specified input to an application.
However, there are many cases where the object hierarchy may not be accessible to an automated testing application. For example, some web development applications (e.g., FLASH®, web 2.0) may not provide useful run time information. Similarly, when an automated testing application accesses applications through an image based protocol (e.g., remote desktop protocol (RDP)) run time information may not be accessible to a computer receiving data via the protocol. This is in part because it is sometimes difficult to identify with which element of a graphical user interface a user is attempting to interact. However, even if a programming language does provide usable run time information, the number of companies using the programming language may be so small that it would not be economical for a company supporting an automated testing application to model the object hierarchy of the programming language.